Coordinating agricultural robots

ABSTRACT

Implementations are described herein for coordinating semi-autonomous robots to perform agricultural tasks on a plurality of plants with minimal human intervention. In various implementations, a plurality of robots may be deployed to perform a respective plurality of agricultural tasks. Each agricultural task may be associated with a respective plant of a plurality of plants, and each plant may have been previously designated as a target for one of the agricultural tasks. It may be determined that a given robot has reached an individual plant associated with the respective agricultural task that was assigned to the given robot. Based at least in part on that determination, a manual control interface may be provided at output component(s) of a computing device in network communication with the given robot. The manual control interface may be operable to manually control the given robot to perform the respective agricultural task.

BACKGROUND

Many agricultural tasks require measures of dexterity and/or delicacythat are difficult for autonomous robots to achieve. For example, it maybe desired, without harming a plant, to trim the plant, to pick weedsthat may have become entangled with the plant, or to pick fruit from theplant. Every plant is unique, which means performance of these taskswill vary extensively between individual plants. Robots arebetter-suited to autonomously perform tasks that are relatively uniformin nature.

SUMMARY

Implementations disclosed herein are directed to coordinatingsemi-autonomous robots to perform agricultural tasks on a plurality ofplants with minimal human intervention. These techniques delegate asmuch of the work as possible to the robots, with a human operatorintervening only if and/or when the robot is incapable of safelyperforming a specific agricultural task with a threshold amount ofconfidence. Techniques described herein allow for a relatively smallnumber of human operators (e.g., as few as one) to control acomparatively large fleet of robots because, e.g., a human operator maymanually control a robot only when the robot reaches a point at which itcannot proceed without posing an unacceptable risk to a plant.

In some implementations, a method performed by one or more processors isprovided that includes: deploying a plurality of robots to perform arespective plurality of agricultural tasks, wherein each agriculturaltask of the plurality of agricultural tasks is associated with arespective plant of a plurality of plants, wherein each plant of theplurality of plants was previously designated as a target for one of theagricultural tasks; determining that a given robot of the plurality ofrobots has reached the individual plant associated with the respectiveagricultural task of the plurality of agricultural tasks that wasassigned to the given robot; based at least in part on the determining,providing, at one or more output components of a computing device innetwork communication with the given robot, a manual control interfacethat is operable to manually control the given robot to perform therespective agricultural task; receiving user input at the manual controlinterface; and operating the given robot in accordance with the userinput received at the manual control interface.

In various implementations, the method may further include rendering aplurality of selectable elements associated with the plurality ofagricultural tasks. In various implementations, each selectable elementof the plurality of selectable elements may be operable to renderadditional information about the respective agricultural task. Invarious implementations, the additional information may include visiondata depicting the plant associated with the respective agriculturaltask. In various implementations, the additional information may furtherinclude one or more annotations overlaying the vision data, wherein theone or more annotations visually emphasize one or more portions of theplant that are to be acted upon by the given robot.

In various implementations, the method may further include: providingoutput in association with one of the robots, wherein the output promptsa user to either: manually control the robot to perform the agriculturaltask assigned to the robot; or predefine a robot plan for the robot tofollow to perform the agricultural task automatically. In variousimplementations, the method may further include deploying one or morescout robots to travel among the plurality of plants and capture sensordata, wherein the sensor data is analyzed to designate each of theplurality of plants as a target for one of the plurality of agriculturaltasks. In various implementations, the one or more scout robots may beseparate from the plurality of robots.

In various implementations, the plurality of agricultural tasks mayinclude one or more of picking a weed from nearby a plant, applying achemical to a plant, picking one or more fruits from a plant, ortrimming a plant.

In another related aspect, a method implemented using one or moreprocessors may include: deploying one or more robots to travel among aplurality of plants and acquire vision data of the plants; analyzing thevision data to designate each of the plurality of plants as a target forperformance of an agricultural task; deploying a plurality of robotsamong the plurality of plants to perform the plurality of agriculturaltasks; and based at least in part on a given robot of the plurality ofrobots arriving at the plant that is the target of the agricultural taskassigned to the given robot, causing a manual control graphical userinterface (“GUI”) to be rendered, wherein the manual control GUI isoperable by a human operator to manually control the given robot toperform the agricultural task assigned to the given robot.

In addition, some implementations include one or more processors (e.g.,central processing unit(s) (CPU(s)), graphics processing unit(s)(GPU(s), and/or tensor processing unit(s) (TPU(s)) of one or morecomputing devices, where the one or more processors are operable toexecute instructions stored in associated memory, and where theinstructions are configured to cause performance of any of theaforementioned methods. Some implementations also include one or morenon-transitory computer readable storage media storing computerinstructions executable by one or more processors to perform any of theaforementioned methods.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically depicts an example environment in which disclosedtechniques may be employed in accordance with various implementations.

FIG. 2 depicts an example field of plants being scouted by multiplerobots to generate agricultural tasks to be performed on the plants.

FIG. 3A, FIG. 3B, FIG. 3C, FIG. 3D, and FIG. 3E depict example graphicaluser interfaces (“GUIs”) that may be interacted with by a human operatorto coordinate performance of the agricultural tasks generated from thescouting of FIG. 2.

FIG. 4 depicts another GUI that may be operable by the human operator toselect between synchronous (manual) and asynchronous (predefined)control of a robot.

FIG. 5 depicts an example GUI that enables asynchronous control of arobot.

FIG. 6 depicts another example GUI that may be interacted with by ahuman operator to coordinate performance of agricultural tasks.

FIG. 7 depicts a flowchart of an example method in accordance withvarious implementations described herein.

FIG. 8 depicts a flowchart of an example method in accordance withvarious implementations described herein.

FIG. 9 schematically depicts an example architecture of a computersystem.

DETAILED DESCRIPTION

Techniques are described herein for coordinating semi-autonomous robotsto perform agricultural tasks on a plurality of plants with minimalhuman intervention. These techniques delegate as much of the work aspossible to the robots, with a human operator intervening only if and/orwhen the robot is incapable of safely performing a specific agriculturaltask with a threshold amount of confidence. Techniques described hereinallow for a relatively small number of human operators (e.g., as few asone) to control a comparatively large fleet of robots because, e.g., ahuman operator may manually control a robot only when the robot reachesa point at which it cannot proceed without posing an unacceptable riskto a plant.

Agricultural tasks such as pruning, picking, weeding, applyingchemicals, deploying beneficial bugs, thinning, seeding, or pollinatingetc., may be identified manually, e.g., by farm workers, orautomatically based on data gathered by robots (in a process that isreferred to herein as “scouting”). In some implementations, one or more“scout” robots equipped with various types of sensors, such astwo-dimensional (“2D”) or three-dimensional (“3D”) vision sensors, maybe deployed to a field to acquire vision data of plants. This visiondata may be analyzed, e.g., by the scout robots themselves or remotelyat one or more computing systems (e.g., one or more computing systemsthat collectively form what is often referred to as the “cloud”), toidentify agricultural tasks for one or more of the plants. For example,one subset of plants may be identified as ready to harvest, anothersubset of plants may be identified as in need of pruning, another subsetmay be identified as being infested by weeds or pests, and so forth (thesubsets may or may not be disjoint). The individual plants of thesesubsets may be flagged for performance of appropriate agriculturaltasks. In some implementations, a scout robot may gather sufficient datato identify specific plant parts that need intervention. As will bediscussed below, these plant parts may be annotated later to aid a humanoperator in causing a robot to perform the agricultural task.

Later, the same robot(s) and/or a different plurality of robots (e.g.,distinct “worker” robots) may be deployed to the field. Each workerrobot may be assigned to perform a particular agricultural task on aparticular plant. Traveling to plants is a relatively easy operation fora robot to perform autonomously, and so the robots may travel to theirrespective plants with little-to-no human intervention. Once a givenrobot arrives at its respective plant, however, it may transition froman autonomous mode to a manual mode. A human operator may then beprovided a manual control graphical user interface (“GUI”) that isoperable by the human operator to control the robot manually while therobot is in manual mode. Once the human manually controls the robot tocomplete the agricultural operation, the robot may transition back intoan autonomous mode and may either move on to another plant to performanother agricultural task, or may return to base.

In some implementations a queue of agricultural tasks may be output forinteraction by human operators. This queue (or more generally, list) ofagricultural tasks may include all of the agricultural tasks describedpreviously, or only some of those agricultural tasks. For example, insome implementations, when a robot autonomously arrives at itsdestination plant and transitions into the manual mode, a newagricultural task may be added to the queue. In some implementations,human operator(s) may commandeer the robots to perform the agriculturaltasks in the order the agricultural tasks appear in the queue.Additionally or alternatively, in some implementations, humanoperator(s) may select whichever agricultural task they want, regardlessof its order in the queue.

While some agricultural tasks are difficult for a robot to performautonomously without posing unacceptable risk to the plant, that is notnecessarily the case for all robot tasks. For example, some agriculturaltasks, such as spraying pesticide or applying fertilizer, may berelatively inexact, and therefore may be performed relatively safely byan autonomous robot, with little or no fear of harming the plant.Accordingly, in some implementations, an agricultural task to beperformed by each robot may be assigned what will be referred to hereinas a “robot performability score” or a “measure of robotperformability.” A robot performability score/measure of robotperformability represents a likelihood that the robot will be able toperform the task autonomously without harming the plant (or itself, oranother robot, or a human).

In some such implementations, agricultural tasks may be presented to thehuman operator in a manner that is selected based on the respectiverobot performability scores. For example, a human operator may bepresented with a plurality of selectable elements, each corresponding toan agricultural task to be performed by a respective robot. Theseselectable elements may or may not be arranged in a queue as describedpreviously. In some implementations, selectable elements may bepresented only for those agricultural tasks with robot performabilityscores that fail to satisfy some threshold (i.e., the agricultural tasksare too difficult or risky for the robot to perform autonomously). Otheragricultural tasks that are less difficult for a robot to performautonomously may proceed without human intervention. In otherimplementations, selectable elements may be presented for allagricultural tasks, but they may be ranked, visually emphasized,color-coded, or otherwise organized in accordance with their respectiverobot performability scores.

It may not be necessary in all cases for a human to manually control arobot synchronously, or in real time. In some implementations, a humanoperator may instead predefine manual robot operations, or a “robotplan,” prior to the robot arriving at its destination. The robot planmay include parameters such as one or more robot arm trajectories to betraversed by an end effector of the robot, one or more grasp poses to beimplemented by the robot's end effector, etc. Once the robot arrives atits destination, it can implement the robot plan autonomously andasynchronously from the manual human input. Meanwhile, the humanoperator may move on to other agricultural tasks assigned to otherrobots, which the human may influence asynchronously by predefining arobot plan or synchronously by manually controlling the robot once itreaches its destination plant.

In some implementations, a human operator may be able to create a robotplan using 2D and/or 3D vision data depicting the plant of interest. Insome implementations, the vision data may be annotated, e.g., based onthe agricultural task, so that the human operator is able to see whatneeds to be done and create a robot plan accordingly. For example, iffruit needs to be picked off a plant, those fruit may be highlighted,and the human operator may create a robot plan by operating a virtualrobot claw/picker to grasp those fruit and remove them from the plant(e.g., by snipping their stems, pulling, etc.). In some suchimplementations, virtual operations of the robot claw/picker controlledby the human operator may be annotated as well, e.g., with lines andcurves to represent trajectories.

In some implementations, the plurality of agricultural tasks may bepresented to a human operator visually, e.g., on a computer display,tablet display, or even as part of a virtual reality (“VR”) or augmentedreality (“AR”) display. In some such implementations, the agriculturaltasks may be represented as a plurality of selectable tiles or icons.These tiles or icons may or may not be ranked, color-coded, animated, orotherwise organized/visually emphasized based on various measures, suchas measures of robot performability discussed previously. For example,tiles representing agricultural tasks that would be difficult for arobot to perform autonomously may be rendered in a first color, tilesrepresenting agricultural task that would be relatively simple for arobot to perform autonomously may be rendered in a second color, andother tiles representing intermediately-difficult agricultural tasks maybe colored in other color(s). In some implementations, tiles may begrouped into clusters based on their relative measures of robotperformability, and those clusters may be rendered on spatially-distinctportions of a display.

Additionally or alternatively, in some implementations, the tiles/iconsrepresenting agricultural tasks may be organized, visually emphasized,color-coded, etc., based on a current state of the robot assigned theagricultural task. For example, a tile representing an agricultural taskfor which a robot is still en route may be rendered in one manner (e.g.,semi-transparently, dashed outline, particular color or shape, etc.)until the robot arrives at the target plant. Once the robot arrives, thetile representing the robot's agricultural task may be visually alteredto inform the human operator that the robot is ready to be controlledmanually/synchronously, if desired. As another non-limiting example, atile representing an agricultural task for which a robot plan has notyet been predefined (and in some cases which has a robot performabilityscore that is conducive to predefinition of a robot plan) may berendered one way, e.g., to inform the human operator that there is stillan opportunity to predefine the robot plan for that robot. Once thehuman operator predefines a robot plan for that task, the correspondingtile may be rendered differently (or demoted in rank, or otherwise putaside) so that the human operator can focus on other tiles for whichasynchronous or synchronous robot control is possible.

FIG. 1 illustrates an environment in which one or more selected aspectsof the present disclosure may be implemented, in accordance with variousimplementations. The example environment includes a plurality of clientdevices 106 _(1−X), an agricultural task system 102, and a plurality ofrobots 108 _(1-M). Each of components 106 _(1−X), 102, and 108 maycommunicate, for example, through a network 110. Agricultural tasksystem 102 is an example of an information retrieval system in which thesystems, components, and techniques described herein may be implementedand/or with which systems, components, and techniques described hereinmay interface.

An individual (which in the current context may also be referred to as a“user”) may operate a client device 106 to interact with othercomponents depicted in FIG. 1. Each component depicted in FIG. 1 may becoupled with other components through one or more networks 110, such asa local area network (“LAN”) or wide area network (“WAN”) such as theInternet. Each client device 106 may be, for example, a desktopcomputing device, a laptop computing device, a tablet computing device,a mobile phone computing device, a computing device of a vehicle of theparticipant (e.g., an in-vehicle communications system, an in-vehicleentertainment system, an in-vehicle navigation system), a standaloneinteractive speaker (with or without a display), or a wearable apparatusthat includes a computing device, such as a head-mounted display (“HMD”)that provides an AR or VR immersive computing experience, a “smart”watch, and so forth. Additional and/or alternative client devices may beprovided.

Each of client devices 106 and agricultural task system 102 may includeone or more memories for storage of data and software applications, oneor more processors for accessing data and executing applications, andother components that facilitate communication over a network. Theoperations performed by client device 106 and/or agricultural tasksystem 102 may be distributed across multiple computer systems. In someimplementations, agricultural task system 102 may be implemented as, forexample, computer program(s) running on one or more computers in one ormore locations that are coupled to each other through a network.

Each client device 106 may operate a variety of different applicationsthat may be used, for instance, to perform agriculture analysis and/orto coordinate agricultural robot operation using techniques describedherein. For example, a first client device 106 ₁ operates agriculturaltask client 107 (e.g., which may be standalone or part of anotherapplication, such as part of a web browser). Another client device 106_(X) may take the form of a HMD that is configured to render 2D and/or3D data to a wearer as part of a VR immersive computing experience. Forexample, the wearer of client device 106 _(X) may be presented with 3Dpoint clouds representing various aspects of objects of interest, suchas fruits of crops. The wearer may interact with the presented data,e.g., using HMD input techniques such as gaze directions, blinks, etc.,to coordinate robots 108 using techniques described herein.

In various implementations, agricultural task system 102 may include avision data analysis engine 112, a robot coordination engine 114, a userinterface engine 116, and/or one or more databases 118 for storingvarious data used by and/or generated by components 112-116, such assensor data gathered by robots 108 _(1−N), agricultural taskinformation, and so forth. In some implementations one or more ofengines 112-116 may be omitted. In some implementations all or aspectsof one or more of engines 112-116 may be combined. In someimplementations, one or more of engines 112-116 may be implemented in acomponent that is separate from agricultural task system 102. In someimplementations, one or more of engines 112-116, or any operativeportion thereof, may be implemented in a component that is executed byclient device 106.

In this specification, the term “database” and “index” will be usedbroadly to refer to any collection of data. The data of the databaseand/or the index does not need to be structured in any particular wayand it can be stored on storage devices in one or more geographiclocations. Thus, for example, database(s) 118 may include multiplecollections of data, each of which may be organized and accesseddifferently.

Vision data analysis engine 112 may be configured to receive or obtain,e.g., from database(s) 118 and/or directly from robots 108 _(1−N), 2Dand/or 3D vision data. In some implementations, vision data analysisengine 112 may receive other robot sensor data as well. This othersensor data may include, for instance, position coordinates such asinertial measurement units (“IMUs”), real-time kinematic (“RTK”)coordinates, GPS coordinates, etc. 2D vision data may be originallyobtained (and then stored in database(s) 118) from various sources. Inthe agricultural context these data may be obtained manually by humanworkers carrying cameras or other sensors, or automatically using one ormore robots 108 _(1−N) equipped with 2D/3D vision sensors. In someimplementations, human workers may simply make observations about plantsusing their own senses, and may manually input those observations into adatabase (e.g., using a spreadsheet or other software application).

Individual robots 108 _(1−N) may take various forms, such as an unmannedaerial vehicle 108 ₁, a robot (not depicted) that is propelled along awire, track, rail or other similar component that passes over and/orbetween crops, wheeled robots 108 _(M−1) to 108 _(M+N), or any otherform of robot capable of being propelled or propelling itself past cropsof interest. In some implementations, different robots may havedifferent roles, e.g., depending on their capabilities. For example, insome implementations, one or more “scout” robots 108 ₁ to 108 _(M−1) maybe designed to capture data, but not necessarily manipulate plants orperform physical agricultural tasks.

By contrast, one or more “worker” robots 108 _(M) to 108 _(M+N) may beconfigured to perform agricultural tasks (and in some cases may alsoserve as scout robots). In some cases, worker robots 108 _(M−N) may bemore complex (and sometimes more costly) than scout robots 108 ₁ to 108_(M−1). For example, in FIG. 1, worker robot 108 _(M) is equipped withone robot arm 120 that includes a gripper-style end effector. Workerrobot 108 _(M+N) is equipped with two robot arms, 120R and 120L. Robotarm 120L is equipped with another gripper-style end effector. Robot arm120R, by contrast, is equipped with a spray end effector 122 that can beoperated, for instance, to spray chemicals such as fertilizer, or evenwater, onto plants.

In some implementations, robots 108 ₁ to 108 _(M−1) (and any availableworker robots 108 _(M) to 108 _(M+N) that are sufficiently capable ofscouting) may scout along lines of crops taking pictures and/orgathering other sensor data (e.g., soil moisture, sunlight readings,temperatures, soil PH, etc.) at some selected frequency (e.g., everysecond or two, every couple of feet, every inch or few inches, etc.).These robots may provide sensor data they capture while scouting toagricultural task system 102, which may store this data in database(s)118.

Based on analysis of the sensor (e.g., vision) data received fromvarious robots 108 while scouting crops and/or from other sources (e.g.,human workers), vision data analysis engine 112 (or robot coordinationengine 114) may generate agricultural tasks and may assign them toindividual robots 108. Vision data analysis engine 112 may performvarious types of image processing analysis to determine variousattributes and/or conditions of plants. These plant attributes and/orconditions may include, but are not limited to, sizes of individualfruit, fruit counts, plant dimensions (e.g., height, width), leafattributes (e.g., count, angle, sizes, etc.), stem count, stem lengths,number of branches, evidence of pests (e.g., mites, slugs, aphids,etc.), plant health (e.g., alone or relative to other nearby plants),plant color, and so forth.

The types of image processing that may be performed by vision dataanalysis engine 112 may vary among implementations. In someimplementations, vision data analysis engine 112 may analyze 2D and/or3D vision data to perform object recognition and/or tracking, which canbe used, for instance, to count and/or track plant-parts-of-interestsuch as fruits, flowers, nuts, berries, and/or other components that mayhave commercial or other value. For object recognition, vision dataanalysis engine 112 may utilize a variety of different techniques,including but not limited to appearance-based methods such as edgematching, divide-and-conquer, gradient matching, greyscale matching,histograms, and/or feature-based methods such as interpretation trees,pose clustering, geometric hashing, scale-invariant feature transform(“SIFT”), speeded up robust features (“SURF”), trained machine learningmodels (e.g., convolutional neural networks), and so forth. Objectrecognition techniques such as trained convolution neural networks mayalso be used to detect other attributes of plants, such as whether theplants are (or have been) infested with/damaged by pests such as aphids,slugs, or mammals (e.g., deer, squirrels, etc.), whether the plants aregenerally healthy (e.g., determined from size relative to othersimilarly-situated plants), whether they are over-watered orunder-watered, etc.

Object tracking is the practice of tracking individual objects (e.g.,fruits) across multiple instances of vision data, such as acrossmultiple 2D digital images of a plant captured by a robot 108 fromdifferent perspectives while the robot travels among the plants. Toperform object tracking, vision data analysis engine 112 may utilizetechniques such as calculating a Euclidian distance between the rawimage data, calculating the Euclidian distance betweenreduced-dimensionality data (e.g., using an encoder portion of ade-noising autoencoder), SIFT, SURF, features from accelerated segmenttest (“FAST”), keypoint matching, object recognition, and so forth. Insome implementations, vision data analysis engine 112 may useplant-parts-of-interest detected across multiple vision files (e.g., 2Ddigital images) to formulate a bipartite graph matching problem that canbe solved, for instance, using techniques such as the Hungarianalgorithm.

In some implementations, vision data analysis engine 112 may have accessto 3D vision data. For example, one or more scout robots 108 ₁ to 108_(M−1) (or even some of worker robots 108 _(M) to 108 _(M+N)) may beequipped with 3D vision sensors such as 3D cameras, light detection andranging (“LIDAR”) sensors, stereo cameras, and/or other sensorssometimes referred to as “2.5D” sensors such as RGBd cameras. In suchimplementations, vision data analysis engine 112 may perform varioustypes of analysis that are possible using 3D vision data, such as objecttracking based on distance between Euclidian coordinates (e.g., how farapart in 3D space are two plant-parts-of-interest detected in 3D data).

However, capturing native 3D vision data can be impractical for variousreasons, such as it being relatively expensive to equip robots with 3Dor 2.5D vision sensors. Accordingly, in some implementations in whichrobots are equipped with less expensive/complex 2D vision sensors, 3Ddata may be generated from multiple 2D digital images using techniquessuch as “structure from motion,” or “SFM” processing. For example, asequence of 2D digital images of plant(s) may be captured, e.g., by arobot moving alongside the plant(s). This sequence of 2D digital imagesmay be analyzed using SFM processing to generate “synthetic” 3D data.This synthetic 3D data may then be used to perform various vision dataanalysis tasks mentioned previously, such as object tracking, objectrecognition, fruit measuring and counting, etc.

Robot coordination engine 114 may be configured to perform selectedaspects of the present disclosure to coordinate deployment of scoutrobot(s) 108 ₁ to 108 _(M−1) and/or worker robots 108 _(M) to 108 _(M+N)acquire sensor data about a plurality of plants (e.g., crops in afield), generate agricultural tasks based on this sensor data, and/or toultimately perform the agricultural tasks. For example, robotcoordination engine 114 may maintain, e.g., in database(s) 118, aninventory of robots 108 _(1−N) that includes information about thoserobots, such as their specifications, capabilities, current state,value, condition, etc.

Based on capabilities of the individual robots 108, robot coordinationengine 114 may assign to individual robots 108 agricultural tasksgenerated by vision data analysis engine 112 and/or generated by othersources, such as agricultural personnel. For example, as part of theprocess of generating agricultural tasks, robot coordination engine 114may identify, from its inventory in database(s) 118, one or more scoutrobots 108 ₁ to 108 _(M−1) and/or one or more worker robots 108 _(M) to108 _(M+N) that are also equipped to perform scouting duties. Robotcoordination engine 114 may then deploy those identified robots totravel among a plurality of plants to gather sensor data about theplants, including 2D and/or 3D vision data of the plants.

These sensor data collected by robots during scouting may be analyzed,e.g., by vision data analysis engine 112 as described previously. Basedon this analysis, vision data analysis engine 112 may designate one ormore individual plants of the plurality of plants as a target forperformance of an agricultural task. Robot coordination engine 114 maythen deploy one or more worker robots 108 _(M) to 108 _(M+N) among theplurality of plants to perform the plurality of agricultural tasks.

As described previously, worker robots 108 _(M) to 108 _(M+N) may nothave sufficient dexterity to perform delicate agricultural tasks such aspicking potentially fragile fruits, pruning branches, removing weeds,etc. Additionally or alternatively, robots may not yet be capable ofperforming these tasks because machine learning model(s) and/or policiesthat govern the robots' operation have not been sufficiently trained.Accordingly, in various implementations, worker robots 108 _(M) to 108_(M+N) may operate more-or-less autonomously until they arrive at theirdestination, i.e., the plant that is the target of the robot's assignedagricultural task. At that point (or in some implementations before therobot arrives), a determination may be made of whether the robot iscapable of performing the task autonomously. If the answer is yes, e.g.,a probability the robot can perform the task without damaging the plantor itself satisfies some threshold, then the robot may be permitted toperform the task itself. However, if the answer is no, then the robotmay transition into a manual control mode in which the robot iscontrolled by a human operator. The human operator may manually controlthe robot via operation one or more client devices 106 _(1−X). Forexample, the human operator may operate agricultural task client 107executing on first client device 106 ₁ to manually control the robot toperform its assigned agricultural task. In situations in which the humanoperator controls the robot “live,” or in real time, that control may bereferred to herein as “synchronous.”

In some implementations, when a given robot arrives at the plant that isthe target of the agricultural task assigned to the given robot, robotcoordination engine 114 may interact with user interface engine 116 tocause a manual control GUI to be provided, e.g., by user interfaceengine 116 to agricultural task client 107. This manual control GUI maybe operable by the human operator to manually control the given robot toperform its agricultural task. Once the human operator operates thegiven robot to complete the task, the given robot may transition to anext agricultural task, which may have been previously assigned to thegiven robot or may be assigned dynamically based on the given robot'scompletion of its previous agricultural task.

For example, unassigned agricultural tasks may be analyzed to identify asubset of one or more candidate agricultural tasks that may be assignedto the given robot based on a variety of factors, such as the givenrobot's capabilities, available resources (e.g., battery power),location, etc. For example, a subset of the agricultural tasks may beidentified for which the given robot is capable of performing. Thatsubset of agricultural tasks may be ranked based on the distancesbetween those candidate agricultural tasks' targeted plants and thegiven robot's current location. The given robot may then be assigned tothe highest-ranked agricultural task.

As noted previously, in some implementations, a human operator may beable to manually control a robot asynchronously instead ofsynchronously. For example, rather than waiting for a robot to arrive atthe plant targeted by the robot's assigned agricultural task, a humanoperator may proactively request a “virtual” manual control interface,e.g., from user interface engine 116. This virtual manual controlinterface is so-named because the human operator may control a virtualrobot to virtually perform the agricultural task ahead of time. Thehuman operator's actions while controlling the virtual robot may be usedto generate what is referred to herein as a “robot plan.” The robot planmay include various robot actions, end effector trajectories, graspposes, etc., that may be implemented by the robot once it arrives at itstarget plant. But by that time, the human operator has been able to moveonto other tasks.

FIG. 2 depicts an example field of plants 232 ₁₋₀ being scouted bymultiple robots 208 ₁₋₂ to generate agricultural tasks to be performedon the plants. First scouting robot 208 ₁ is moving in the downwarddirection and is gathering vision data about a first row of plants 232₁₋₅ using its vision sensor 209 ₁. Second scouting robot 208 ₂—which mayalso be used as a working robot since it has a robot arm 220—is movingin the downward direction and is gathering vision data about a secondrow of plants 232 ₆₋₁₀ using its vision sensor 209 ₂.

From the vision data gathered by robots 208 ₁₋₂, vision data analysisengine 112 may make various observations about plants 232 ₁₋₁₀. Theseobservations can be used by vision data analysis engine 112 (or anothercomponent of agricultural task system 102) to generate agriculturaltasks. For example, second plant 232 ₂, fourth plant 232 ₄, seventhplant 232 ₇, and ninth plant 232 ₉ appear ready for harvesting becausethey have multiple fruits that are relatively large. By contrast, firstplant 232 ₁ and third plant 232 ₃ are not ready for harvesting becausetheir fruit are less numerous and/or smaller. Additionally, fifth plant232 ₅ and tenth plant 232 ₁₀, which are proximate each other acrossrows, are being overtaken and/or entangled by weeds. The same is truefor sixth plant 232 ₆. Eighth plant 232 ₈ has been killed off entirely,leaving only a weed in its place.

From these observations, a plurality of agricultural tasks may begenerated, e.g., by vision data analysis engine 112 and/or robotcoordination engine 114. At least some of these agricultural tasksassociated with the first row of plants 232 ₁₋₅ are manifested asoperable elements (e.g., selectable tiles) 350 ₁₋₅ in FIG. 3A. Operableelements 350 ₁₋₅ in particular are presented by an example GUI 360entitled “2019 HYPOTHETICAL FARMS AGRICULTURAL TASK LIST.” GUI 360 mayor may not be rendered on one or more client devices 106, e.g., as partof agricultural task client 107.

First operable element 350 ₁ is associated with an agricultural taskgenerated based on observations of first plant 232 ₁. As notedpreviously, first plant 232 ₁ is not ready for harvest. Accordingly,first operable element 350 ₁ is associated with task 00001, “Sprayw/fertilizer.” As noted in first operable element 350 ₁, this task is tobe performed automatically (autonomously) by a robot that is currentlyen route to first plant 232 ₁. Second operable element 350 ₂ isassociated with second plant 232 ₂, which is ready for harvest.Accordingly, second operable element 350 ₂ is associated with task00002, “harvest fruit,” which is to be performed manually “upon arrival”of a robot to second plant 232 ₂.

Third operable element 350 ₃ is associated with third plant 232 ₃, whichis not ready for harvest but is in need of pruning. Accordingly, thirdoperable element 350 ₃ is associated with task 00003, “prune,” which isto be performed manually “upon arrival” of a robot to third plant 232 ₃.Fourth operable element 350 ₄ is associated with fourth plant 232 ₄,which is ready for harvest. Accordingly, fourth operable element 350 ₄is associated with task 00004, “harvest fruit,” which is to be performedmanually “upon arrival” of a robot to fourth plant 232 ₄. And fifthoperable element 350 ₅ is associated with fifth plant 232 ₅, which isbeing overtaken by weeds. Accordingly, fifth operable element 350 ₅ isassociated with task 00005, “pick weed(s),” which is to be performedmanually “upon arrival” of a robot to fifth plant 232 ₅.

In FIG. 3A, the robots assigned the agricultural tasks associated withoperable elements 350 ₁₋₅ have only just been deployed, and have not yetreached their respective target plants. Accordingly, each of operableelements 350 ₁₋₅ indicates a current status on the bottom line of “enroute.” In some implementations, the assigned robots cannot beinteracted with or controlled until they reach their target plants.Assuming that's the case in this example, operable elements 350 ₁₋₅ areshaded to indicate that they may not yet be operable, or at least notoperable to commandeer control of the robots.

In FIG. 3B, by contrast, operable elements 350 ₁₋₃ are no longer shadedbecause the respective robots assigned the underlying agricultural taskshave arrived at the target plants. Recall that first operable element350 ₁ is associated with an agricultural task, “spray w/fertilizer,”that is intended to be performed by a robot automatically. Accordingly,because that robot has since arrived at the plant (232 ₁) targeted bythe agricultural task 00001 underlying first operable element 350 ₁, therobot has begun applying the fertilizer, as indicated by its status,“applying fertilizer.” In some cases a human operator could select firstoperable element 350 ₁ if, for instance, the human operator wanted towatch real-time application of the fertilizer (e.g., in a viewfinderthat renders vision data captured by a vision sensor integral with thatrobot), or if the human operator wanted to commandeer control of therobot to apply the fertilizer manually. But otherwise the human operatorcan take no action, and once the robot has completed its application offertilizer to first plant 232 ₁, that robot may be assigned anotheragricultural task.

Second and third operable elements 350 ₂₋₃, on the other hand, bothindicate a state of “awaiting instructions” because their respectiveunderlying tasks, “harvest fruit” and “prune,” require manualintervention by a human operator. A human operator may select either ofthese operable elements 350 ₂₋₃ to gain control of the respective robotassigned the underlying agricultural task.

Suppose a human operator selects second operable element 350 ₂. In someimplementations, the human operator may be presented with the GUI 362depicted in FIG. 3C. GUI 362 includes a window 364 that depicts visiondata captured by a vision sensor of a robot assigned the agriculturaltask (00002) underlying second operable element 350 ₂ in FIG. 3B.Consequently, second plant 232 ₂ is visible in window 364. Also visibleare two robot arms, 320L and 320R, which the human operator can remotecontrol manually to pick fruit from second plant 232 ₂.

In some implementations, one or more components of agricultural tasksystem 102, such as vision data analysis engine 112, may applyannotations to vision data based on its analysis of vision data receivedduring prior scouting of a field of plants. These annotations may aid ahuman operator in manually controlling a robot to perform anagricultural task. An example of this is depicted in FIG. 3C, where fivevisual annotations 366 ₁₋₅ are depicted in the form of bounding shapesaround five different fruit of second plant 232 ₂. These bounding shapesmay have been detected in vision data gathered previously, e.g., byscouting robots 108 ₁ to 108 _(M−1). These annotations 366 ₁₋₅ may helpthe human operator 232 understand what needs to be done to fulfill theagricultural task. In this example, that task is picking fruit, and theannotations 366 ₁₋₅ call out where the fruit is that needs to be picked.While annotations 366 ₁₋₅ are depicted as circles, this is not meant tobe limiting, and bounding shapes used for such purposes may take anynumber of forms.

FIG. 3C to control a robot to perform the agricultural task 00002. Forexample, the human operator may be working on a client device 106 thatincludes multi-window capabilities (e.g., as part of an operatingsystem), and GUIs 360 and 362 may be rendered on different parts of thescreen, or on different screens in a multi-screen environment.Alternatively, the human operator may be able to switch between GUIs 360and 362.

In FIG. 3D, operable elements 350 ₂₋₅ are now unshaded because all ofthe robots assigned their underlying agricultural tasks have arrived attheir respective target plants. And as shown in FIG. 3D, operableelements 350 ₃₋₅ indicate a current status of “awaiting instructions.”Second operable element 350 ₂ shows a status of “under manual control”because as noted previously, the human operator has not yet finishedcontrolling the robot to pick fruit from second plant 232 ₂. In someimplementations, the human operator can leave task 00002 open and switchto another task, such as any of tasks 00003-000005 underlying operableelements 350 ₃₋₅, by selecting any one of those operable elements 350₃₋₅. The robot associated with second operable element 350 ₂ may simplystandby until the human operator “returns,” e.g., by clicking secondoperable element 350 ₂.

First operable element 350 ₁ has been re-shaded because its underlyingagricultural task 00001 has been “completed.” Consequently, the currentstatus of first operable element 350 ₁ indicates a next agriculturaltask, 00006, that has been assigned to the robot. As will be seenshortly, once that robot has departed for its next task, first operableelement 350 ₁ may be replaced with a new operable element (350 ₆ in FIG.3E) that represents the new agricultural task assigned to that robot.

FIG. 3E depicts GUI 360 at some later time after the human operator hasfinished interacting with GUI 362 (FIG. 3C) to control the robot toperform the agricultural task 00002 underlying second operable element350 ₂, which now indicates that task 00002 is completed. A currentstatus conveyed by second operable element 350 ₂ is that the robot,which was manually operated to complete that task 00002, has now beenassigned a new task 00007.

Additionally, in FIG. 3E, a new operable element 350 ₆ is rendered thatidentifies a new task 00006 that has been assigned to the robot thatpreviously performed the agricultural task 00001. For the task 00006,the robot is supposed to be manually operated upon arrival at sixthplant 232 ₆ in FIG. 2 to “pick weed(s).” This task may have beenassigned to that robot because the target plant (232 ₆) is the closestto the plant (232 ₁) previously operated on by the robot (when itpreviously sprayed fertilizer onto first plant 232 ₁).

In FIGS. 3A-B and D-E, the operable elements 350 ₁₋₅ are presented moreor less in the order that the first row of plants 232 ₁₋₅ were scoutedby robot 208 ₁ in FIG. 2. However, this is not meant to be limiting.Operable elements in general may be rendered on user interfaces indifferent orders and/or in manners that are selected based on a varietyof signals. For example, and as discussed previously, agricultural tasksmay be assigned, e.g., by vision data analysis engine 112, robotperformability scores, or measures of robot performability. Measures ofrobot performability may indicate how likely a robot will be able toautonomously perform the agricultural tasks without posing unacceptablerisk(s) to the plant, or to other robots/equipment. These measures ofrobot performability may be used, for instance to rank operableelements. As a result, the human operator's attention is focused onthose agricultural tasks which are most likely to require humanintervention.

Additionally or alternatively, in some implementations, operableelements may be rendered only if their underlying agricultural taskshave measures of robot performability that fail to satisfy some minimalthreshold. In other words, operable elements may not be rendered at or(or at least may be rendered less prominently) for agricultural tasksthat a robot is likely to be able to perform without much risk ofharming itself or the plant. These agricultural tasks typically requireless dexterity, and may include tasks such as spraying a plant withchemicals/fertilizer, picking an entire plant from the ground (e.g.,which may be the case with plants such as lettuce or cabbage), wateringa plant, releasing anti-pest agents such as aphid-eating beneficial bugs(e.g., ladybugs) near a plant, and so forth.

A measure of robot performability may be determined, e.g., by visiondata analysis engine 112, in various ways based on various data points.For example, in some implementations, particular agricultural tasks maybe preassigned robot performability scores, e.g., in a database orlookup table. These preassigned scores may be determined, for instance,based on empirical evidence and/or based on observations fromagricultural and/or robotic personnel.

Additionally or alternatively, in some implementations, machine learningmodel(s) may be trained to predict, e.g., based on vision data of aplant, a measure of robot performability associated with an agriculturaltasks. In some implementations, supervised, semi-supervised, or evenunsupervised learning may be employed to train, for example, aconvolutional neural network (“CNN”) to predict such measures. The CNNmay be trained with training examples that depict various plants invarious poses/configurations. In some implementations, the trainingexamples may be labeled with measures of robot performability. Theselabels may be compared to output of the CNN generated based on thetraining examples to optimize a loss function of the CNN, e.g., usingtechniques such a gradient descent, back propagation, etc. The labelsmay be assigned to the training examples by humans, or based on priorrobot attempts to perform agricultural tasks on the plants depicted inthe training data.

For example, suppose a robot damaged a plant depicted in a trainingexample when the robot attempted to autonomously perform an agriculturaltask on the plant. That training example may be used as a negativetraining example, e.g., with its label being a low robot performabilityscore, or simply a binary value indicating failure. By contrast, supposeanother robot successfully and autonomously performed an agriculturaltask on another plant depicted in another training example. That secondtraining example may be used as a positive training example, e.g., withits label being a relatively high robot performability score, or simplya binary value indicating success.

As noted previously, in some implementations, human operators are notlimited to synchronous control of robots after those robots have arrivedat their target plants. In some such implementations, human operatorsmay be able to predefine robot plans that can be implemented by robotslater, e.g., in an asynchronous manner. FIG. 4 depicts another GUI 468that may be operable by the human operator to select between synchronous(manual) and asynchronous (predefined) control of a robot. GUI 468 maybe presented to a human operator when, for instance, the human operatorselects an operable element 350 ₂ before the corresponding robot arrivesat its destination plant (232 ₂).

If the human operator selects the “manual control” button of GUI 468, insome implementations, GUI 360 may regain primary focus on the user'sscreen, and operable element 350 ₂ that the human operator previouslyclicked may be available for selection, e.g., once the robot arrives atthe target plant (232 ₂). On the other hand, if the human operatorselects the “predefine plan” button of GUI 468, the human operator maybe presented with a “virtual control” or “asynchronous control” GUI 470depicted in FIG. 5.

Virtual control GUI 470 in FIG. 5 is similar to GUI 362 in FIG. 3C. Thesame plant 232 ₂ and annotations 366 ₁₋₅ that were depicted in FIG. 3Care depicted in FIG. 4, for instance. However, rather than depictingphysical robot arms 320L and 320R as detected by a vision sensor, inFIG. 5, virtual control GUI 470 depicts virtual arms 320Lv and 320Rv, asrepresented by the dashed lines. Virtual arms 320Lv and 320Rv may becontrolled by a human operator to perform the task (picking fruit inaccordance with annotations 366 ₁₋₅) in a similar fashion as in FIG. 3C.However, because the robot has not actually arrived at second plant 232₂ yet, the human operator's inputs may be recorded prospectively, e.g.,as a robot plan.

This robot plan, which may include end effector trajectories, graspposes, etc., can be implemented by the robot later, after the robotarrives at plant 232 ₂. In FIG. 5, for instance, a trajectory 380L ofvirtual robot arm 320Lv and a trajectory 380R of virtual robot arm 320Rvare depicted. These trajectories 380L, 380R travel amongst the variousfruits. They may be recorded, along with grasp poses implemented tograsp the fruits, and then implemented automatically by the robot laterafter the robot actually arrives at plant 232 ₂.

There is some risk that a plant may change between the time it isscouted, the time a human operator prospectively creates a robot planusing virtual control GUI 470, and the time the robot actually arrives.For example, one or more fruit may have fallen off, the wind may haveblown the fruit to different locations, the plant may have been damaged(accidentally by agricultural personnel or otherwise by pest oranimals), etc. Accordingly, in some implementations, various checks maybe performed by the robot once it arrives at the plant to ensure therobot plan is still viable.

For example, in some implementations, the vision data captured duringscouting and the vision data captured by the robot once it arrives atplant 232 ₂ may be compared to ensure that plant 232 ₂ has not changedso much that the robot plan would no longer be effective. In some suchimplementations, some threshold difference in appearance or similarity(e.g., Euclidian distance between raw data) between the two instances ofvision data may trigger abandonment of the robot plan. In such ascenario, a human operator may be prompted to perform the task manually,and the robot plan may be discarded.

The interfaces depicted in FIGS. 3A-E, 4, and 5 are not meant to belimiting. Any number of other interfaces may be used instead. Forexample, GUIs 360, 362, 468, and 470 take the form of windows that mightbe rendered on a computer display, tablet computer, mobile devicescreen, etc. Other types of interfaces might be rendered on other typesof displays, such as display(s) integral with a HMD. For example,operable elements may be rendered as 3D objects that a human operatorwearing the HMD can select using, for instance, their gaze, by blinking,by issuing voice commands, etc.

In some implementations, interfaces rendered/provided/generated by userinterface engine 116 may focus on the robots themselves, rather than ontheir assigned agricultural tasks. Once such example is GUI 688 in FIG.6. GUI 688 is similar to GUI 360 in FIG. 3A in many respects. However,rather than operable elements 350 ₁₋₅ corresponding to agriculturaltasks 00001-00005, with GUI 688, a plurality of operable elements 690₁₋₅ correspond to five different robots named “Archie,” “Carl,”“Redmond,” “Delia,” and “Mavis.”

Similar to FIG. 3A, each robot has been assigned a particularagricultural task. For example, Archie is assigned task 00001 of “sprayw/fertilizer,” Redmond has been assigned the task 00003 of “prune,” andso forth. And each operable element 690 indicates a status of the robot,which is currently “en route” for all five robots. Similar to before, ahuman operator may click on an operable element 690 to view moreinformation about the agricultural task to which the selected robot isassigned, or to predefine a robot plan to be implemented by the robotupon its arrival at a target plant. Or, once a given robot arrives atits target plant, the robot's corresponding operable element 690 may beupdated so that its status reads something like, “awaitinginstructions.” The human operator could then click that operable element690 to open an interface similar to that depicted in FIG. 3C.

FIG. 7 illustrates a flowchart of an example method 700 for practicingselected aspects of the present disclosure. The operations of FIG. 7 canbe performed by one or more processors, such as one or more processorsof the various computing devices/systems described herein, such as byagricultural task system 102. For convenience, operations of method 700will be described as being performed by a system configured withselected aspects of the present disclosure. Other implementations mayinclude additional steps than those illustrated in FIG. 7, may performstep(s) of FIG. 7 in a different order and/or in parallel, and/or mayomit one or more of the steps of FIG. 7.

At block 702, the system, e.g., by way of robot coordination engine 114,may deploy a plurality of robots, such as worker robots 108 _(M) to 108_(M+N), to perform a respective plurality of agricultural tasks. Eachagricultural task of the plurality of agricultural tasks may beassociated with a respective plant of a plurality of plants, e.g., in afield. And each plant of the plurality of plants may have beenpreviously designated as a target for one of the agricultural tasks,e.g., by agricultural task system 102.

At block 704, the system may determine that a given robot of theplurality of robots has reached the individual plant associated with therespective agricultural task of the plurality of agricultural tasks thatwere assigned to the given robot. For example, the robot may send atransmission to agricultural task system 102 and/or robot coordinationengine 114 that signals its arrival at the plant that is the target ofthe robot's agricultural task.

Based at least in part on determining at block 704 that the given robothas arrived at its target plant, at block 706, the system may determinewhether the robot is capable of performing the agricultural task withoutharming the plant, the plant's commercial produce, or the robot itself.For example, in some implementations, a probability that the robot willbe able to perform its assigned agricultural task without harming theplant or its produce may be determined. If that probability satisfiessome threshold, the robot may be allowed to perform the task. In someimplementations, there may be a lesser range of probabilities thatindicate a robot is likely going to be able to perform the task, butwith somewhat less confidence. In some such implementations, a humanoperator may be given a choice as to whether to allow the robot toperform the agricultural task autonomously or to control the robotremotely to perform the task. The determination at block 706 does notnecessarily have to be performed once the robot arrives at the plant. Insome cases this determination may be made as soon as the agriculturaltask is assigned to a particular robot (in which case the robot'scapabilities will be known).

If it is determined at block 706 that the robot can perform theagricultural task autonomously, then method 700 may proceed to block708, and the robot may be allowed to perform the agricultural taskautonomously. However, if it is determined at block 706 that the robotcannot (at least with an acceptable measure of confidence) perform theagricultural task autonomously, then method 700 may proceed to block710.

At block 710, the system may provide, at one or more output componentsof any one of client devices 106 _(1-X) that are in networkcommunication (e.g., via 110) with the given robot, a manual controlinterface that is operable to manually control the given robot toperform the respective agricultural task. One example of such a manualcontrol interface was depicted at 362 in FIG. 3C.

At block 712, the system may receive user input via the manual controlinterface. For example, the human operator may operate one or more inputdevices, such as a keyboard, mouse, joystick(s), touchscreen, etc., toprovide inputs to control robot arm(s), end effectors, etc. At block714, the system may operate the given robot in accordance with the userinput received at the manual control interface. For example, the userinput may be relayed directly to the given robot, or may be processed,e.g., by robot coordination engine 114, and then sent to the givenrobot.

FIG. 8 illustrates a flowchart of an example method 800 for practicingselected aspects of the present disclosure. The operations of FIG. 8 canbe performed by one or more processors, such as one or more processorsof the various computing devices/systems described herein, such as byagricultural task system 102. For convenience, operations of method 800will be described as being performed by a system configured withselected aspects of the present disclosure. Other implementations mayinclude additional steps than those illustrated in FIG. 8, may performstep(s) of FIG. 8 in a different order and/or in parallel, and/or mayomit one or more of the steps of FIG. 8.

At block 802, the system, e.g., by way of robot coordination engine 114,may deploy one or more robots, such as one or more scout robots 108 ₁ to108 _(M−1), to travel among a plurality of plants and acquire visiondata of the plants. As noted previously, any robot equipped with visionsensor(s) may potentially be used as a scout robot, regardless ofwhether that robot is also used later as a worker robot to perform anagricultural task.

At block 804, the system, e.g., by way of vision data analysis engine112, may analyze the vision data gathered at block 802 to designate eachof the plurality of plants as a target for performance of anagricultural task. As used herein, “to designate each of the pluralityof plants” does not necessarily mean that every single plant in a givenfield will be assigned an agricultural task. The vision data gathered atblock 802 may reveal, for instance, that some plants are not currentlyin need of intervention.

At block 806, the system, e.g., by way of robot coordination engine 114,may deploy a plurality of robots among the plurality of plants toperform the plurality of agricultural tasks. The operations of block 806may be similar to the operations of block 702 of method 700. At block808, a determination may be made, e.g. by robot coordination engine 114,whether any of the robots deployed at block 806 has reached a targetplant. This determination may be based on, for example, a signal from arobot indicating that it has arrived at its target plant and is awaitingfurther instructions.

Based at least in part on a determination at block 808 that a givenrobot of the plurality of robots deployed at block 806 has arrived atthe plant that is the target of the agricultural task assigned to thegiven robot, at block 810, the system may cause a manual control GUI tobe rendered on a client device 106. As was the case previously, themanual control GUI may be operable by a human operator to manuallycontrol the given robot to perform the agricultural task assigned to thegiven robot. In some implementations, the manual control GUI may not beprovided immediately, but instead may be made available to the humanoperator, e.g., as a tile or icon that is selectable to launch themanual operation GUI.

FIG. 9 is a block diagram of an example computing device 910 that mayoptionally be utilized to perform one or more aspects of techniquesdescribed herein. Computing device 910 typically includes at least oneprocessor 914 which communicates with a number of peripheral devices viabus subsystem 912. These peripheral devices may include a storagesubsystem 924, including, for example, a memory subsystem 925 and a filestorage subsystem 926, user interface output devices 920, user interfaceinput devices 922, and a network interface subsystem 916. The input andoutput devices allow user interaction with computing device 910. Networkinterface subsystem 916 provides an interface to outside networks and iscoupled to corresponding interface devices in other computing devices.

User interface input devices 922 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In some implementations in which computingdevice 910 takes the form of a HMD or smart glasses, a pose of a user'seyes may be tracked for use, e.g., alone or in combination with otherstimuli (e.g., blinking, pressing a button, etc.), as user input. Ingeneral, use of the term “input device” is intended to include allpossible types of devices and ways to input information into computingdevice 910 or onto a communication network.

User interface output devices 920 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, one or more displays forming part of a HMD, or some othermechanism for creating a visible image. The display subsystem may alsoprovide non-visual display such as via audio output devices. In general,use of the term “output device” is intended to include all possibletypes of devices and ways to output information from computing device910 to the user or to another machine or computing device.

Storage subsystem 924 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 924 may include the logic toperform selected aspects of the method described herein, as well as toimplement various components depicted in FIG. 1.

These software modules are generally executed by processor 914 alone orin combination with other processors. Memory 925 used in the storagesubsystem 924 can include a number of memories including a main randomaccess memory (RAM) 930 for storage of instructions and data duringprogram execution and a read only memory (ROM) 932 in which fixedinstructions are stored. A file storage subsystem 926 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 926 in the storage subsystem 924, or inother machines accessible by the processor(s) 914.

Bus subsystem 912 provides a mechanism for letting the variouscomponents and subsystems of computing device 910 communicate with eachother as intended. Although bus subsystem 912 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computing device 910 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computing device 910depicted in FIG. 9 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputing device 910 are possible having more or fewer components thanthe computing device depicted in FIG. 9.

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

What is claimed is:
 1. A method implemented using one or moreprocessors, comprising: deploying a plurality of robots to perform arespective plurality of agricultural tasks, wherein each agriculturaltask of the plurality of agricultural tasks is associated with arespective plant of a plurality of plants, wherein each plant of theplurality of plants was previously designated as a target for one of theagricultural tasks; for each agricultural task of the plurality ofagricultural tasks, calculating a measure of robot performability thatrepresents a likelihood that the respective robot of the plurality ofrobots assigned to the agricultural task will be able to perform theagricultural task successfully; causing a graphical user interface (GUI)to be rendered on a display, wherein the GUI includes a plurality ofselectable elements corresponding to the plurality of agriculturaltasks, and wherein each of the selectable elements is selectable toinfluence performance of the corresponding agricultural task by therespective robot of the plurality of robots assigned to the agriculturaltask; and causing a given selectable element of the plurality ofselectable elements to be rendered to convey an indication of themeasure of robot performability calculated for the agricultural taskrepresented by the given selectable element.
 2. The method of claim 1,further comprising: determining that a given robot of the plurality ofrobots has autonomously reached the individual plant of the plurality ofplants that is associated with the respective agricultural task of theplurality of agricultural tasks that was assigned to the given robot;and in response to the determining, causing the selectable elementrepresenting the respective agricultural task to convey a status for thegiven robot assigned to the respective agricultural task, wherein thestatus comprises awaiting manual control or performing the respectiveagricultural task.
 3. The method of claim 2, further comprising: inresponse to selection of the selectable element representing therespective agricultural task, causing the GUI to render a manual controlinterface that is operable to manually control the given robot toperform the respective agricultural task; receiving user input at themanual control interface; and operating the given robot in accordancewith the user input received at the manual control interface.
 4. Themethod of claim 1, further comprising causing the given selectableelement to be rendered with a previously-acquired image of the planttargeted by the agricultural task represented by the given selectableelement.
 5. The method of claim 4, wherein the previously-acquired imageis annotated to identify one or more plant-parts-of-interest to be actedupon by the robot assigned to the agricultural task represented by thegiven selectable element.
 6. The method of claim 5, wherein the givenselectable element is selectable to asynchronously predefine a robotplan to be implemented by the robot assigned to the agricultural taskrepresented by the given selectable element.
 7. The method of claim 6,further comprising generating a trajectory of an end effector of therobot assigned to the agricultural task represented by the givenselectable element, wherein the generating is based on interaction atthe GUI with the annotated plant-parts-of-interest of thepreviously-acquired image.
 8. A system comprising one or more processorsand memory storing instructions that, in response to execution of theinstructions, cause the one or more processors to: deploy a plurality ofrobots to perform a respective plurality of agricultural tasks, whereineach agricultural task of the plurality of agricultural tasks isassociated with a respective plant of a plurality of plants, whereineach plant of the plurality of plants was previously designated as atarget for one of the agricultural tasks; for each agricultural task ofthe plurality of agricultural tasks, calculate a measure of robotperformability that represents a likelihood that the respective robot ofthe plurality of robots assigned to the agricultural task will be ableto perform the agricultural task successfully; cause a graphical userinterface (GUI) to be rendered on a display, wherein the GUI includes aplurality of selectable elements corresponding to the plurality ofagricultural tasks, and wherein each of the selectable elements isselectable to influence performance of the corresponding agriculturaltask by the respective robot of the plurality of robots assigned to theagricultural task; and cause a given selectable element of the pluralityof selectable elements to be rendered to convey an indication of themeasure of robot performability calculated for the agricultural taskrepresented by the given selectable element.
 9. The system of claim 8,further comprising instructions to: determine that a given robot of theplurality of robots has autonomously reached the individual plant of theplurality of plants that is associated with the respective agriculturaltask of the plurality of agricultural tasks that was assigned to thegiven robot; and in response to the determination that the given robothas autonomously reached the individual plant, cause the selectableelement representing the respective agricultural task to convey a statusfor the given robot assigned to the respective agricultural task,wherein the status comprises awaiting manual control or performing therespective agricultural task.
 10. The system of claim 9, furthercomprising instructions to: in response to selection of the selectableelement representing the respective agricultural task, cause the GUI torender a manual control interface that is operable to manually controlthe given robot to perform the respective agricultural task; receiveuser input at the manual control interface; and operate the given robotin accordance with the user input received at the manual controlinterface.
 11. The system of claim 8, further comprising instructions tocause the given selectable element to be rendered with apreviously-acquired image of the plant targeted by the agricultural taskrepresented by the given selectable element.
 12. The system of claim 11,wherein the previously-acquired image is annotated to identify one ormore plant-parts-of-interest to be acted upon by the robot assigned tothe agricultural task represented by the given selectable element. 13.The system of claim 12, wherein the given selectable element isselectable to asynchronously predefine a robot plan to be implemented bythe robot assigned to the agricultural task represented by the givenselectable element.
 14. The system of claim 13, further comprisinginstructions to generate a trajectory of an end effector of the robotassigned to the agricultural task represented by the given selectableelement, wherein the trajectory is generated based on interaction at theGUI with the annotated plant-parts-of-interest of thepreviously-acquired image.
 15. A non-transitory computer-readable mediumcomprising instructions that, in response to execution of theinstructions by a processor, cause the processor to: deploy a pluralityof robots to perform a respective plurality of agricultural tasks,wherein each agricultural task of the plurality of agricultural tasks isassociated with a respective plant of a plurality of plants, whereineach plant of the plurality of plants was previously designated as atarget for one of the agricultural tasks; for each agricultural task ofthe plurality of agricultural tasks, calculate a measure of robotperformability that represents a likelihood that the respective robot ofthe plurality of robots assigned to the agricultural task will be ableto perform the agricultural task successfully; cause a graphical userinterface (GUI) to be rendered on a display, wherein the GUI includes aplurality of selectable elements corresponding to the plurality ofagricultural tasks, and wherein each of the selectable elements isselectable to influence performance of the corresponding agriculturaltask by the respective robot of the plurality of robots assigned to theagricultural task; and cause a given selectable element of the pluralityof selectable elements to be rendered to convey an indication of themeasure of robot performability calculated for the agricultural taskrepresented by the given selectable element.
 16. The non-transitorycomputer-readable medium of claim 15, further comprising instructionsto: determine that a given robot of the plurality of robots hasautonomously reached the individual plant of the plurality of plantsthat is associated with the respective agricultural task of theplurality of agricultural tasks that was assigned to the given robot;and in response to the determination that the given robot hasautonomously reached the individual plant, cause the selectable elementrepresenting the respective agricultural task to convey a status for thegiven robot assigned to the respective agricultural task, wherein thestatus comprises awaiting manual control or performing the respectiveagricultural task.
 17. The non-transitory computer-readable medium ofclaim 16, further comprising instructions to: in response to selectionof the selectable element representing the respective agricultural task,cause the GUI to render a manual control interface that is operable tomanually control the given robot to perform the respective agriculturaltask; receive user input at the manual control interface; and operatethe given robot in accordance with the user input received at the manualcontrol interface.
 18. The non-transitory computer-readable medium ofclaim 15, further comprising instructions to cause the given selectableelement to be rendered with a previously-acquired image of the planttargeted by the agricultural task represented by the given selectableelement.
 19. The non-transitory computer-readable medium of claim 18,wherein the previously-acquired image is annotated to identify one ormore plant-parts-of-interest to be acted upon by the robot assigned tothe agricultural task represented by the given selectable element. 20.The non-transitory computer-readable medium of claim 19, wherein thegiven selectable element is selectable to asynchronously predefine arobot plan to be implemented by the robot assigned to the agriculturaltask represented by the given selectable element.